**************************************** RELEASE NOTES **************************************** Chelsio Unified Wire for Linux ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ CHANGES ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Driver ====== Version : 3.19.0.1 Date : 01/22/2024 ================================================================================ - Updated Firmware to 1.27.5.0 - Added support for Distributions RHEL/Rocky/AlmaLinux 9.3 and 8.9, Debian 12.4, kernel.org 6.6.X kernels. Network (NIC/TOE): - Fixed the compilation issues observed on latest 6.1.X, 5.15.X kernels. iWARP RDMA Offload: - Added support for rping loopback test over vlan for IPv6. SPDK NVMe-oF TOE: - Added support for SPDK v23.01.1 LTS. ================================================================================ ================================================================================ Version : 3.18.0.1 Date : 08/04/2023 ================================================================================ - Updated Firmware to 1.27.4.0 - Added support for Distributions RHEL/Rocky Linux 9.2 and 8.8 Crypto Offload: - Fixed the low bandwidth issue with Inline TLS TOE traffic. - Fixed a kernel panic with coprocessor when iommu is enabled. LIO iSCSI Target Offload: - Have set the default values for TCP Send and Receive window sizes. Firmware ======== Version : 1.27.5.0 Date : 10/10/2023 ================================================================================ Fixes ----- BASE: - Fixed handling the Remote Fault with AN, causing the link failure. ================================================================================ ================================================================================ Version : 1.27.4.0 Date : 07/05/2023 ================================================================================ Fixes ----- BASE: - Handle 40G to 100G cable change. - Avoid unnecessary i2c read. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ERRATA (Open Issues) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Network (NIC/TOE) ----------------- - 43831: [RHEL9.3]: Ping fails on running TOE netperf TCP_RR connections. - 43132: cxgbtool fec auto is not using cable/transceiver recommended FEC in few cases. - 42666: ethtool 'rx-vlan-offload off' is not taking effect. - 42430: [RHEL8.6]: netlink error observed while trying to read the eeprom of 100g ports. - 41528: Mismatch in "Port Capabilities" seen in dmesg on unplugging 1g modules and inserting 10g modules. - 41150: Link fails to come up on one of the ports with repeated driver reloads, when 40G cables are used on both ports. - 40948: SEND-Q values under "ss -tin" show negative values for TOE connections. - 40425: Kernel panic with continuous flood of syn packets to offloaded server. - 40000: [TOE]: Need support for asynchronous socket mechanism. - 38844/36020: 100G link fails to come up with few switches with FEC OFF and AN ON. Workaround: Disable AN for the link to come up. - 37930: Need support for ethtool -l. - 35372: ethtool does not report correct values on swapping the fibre with twinax cables. - 35353: Rx packets are getting dropped when double VLAN is configured. - 35243: TCP Fin and Ack packets are seen in tcpdump while running large number of TOE connections. - 34592: Link goes down on plug/unplug with AN off QSFP28/SFP28 Fibre/Copper. - 34179: Setting 1G speed for T6 optics is not supported. - 34142: Trace observed while unloading t4_tom on RHEL6.X platforms. NOTE: This is a kernel issue and will be seen only with multiple adapters. - 34068: RS FEC is not supported on 50G link. - 33643: [VXLAN]: No support for computing outer UDP checksum. - 33531: LED blinks rapidly on 100G adapters when we run ethtool -p ethX and connect the Twinax cable. - 32844: devlog is flooded with link related messages when 1G link is not UP. - 32825: T5 40G twinax link fails to come up when connected to few switches. - 32515: [PTP] Unable to start 'HW time stamping' master clock with Network transport option. - 31775: deadman_watchdog timer is not accurate for mode=pauseoff. - 27853: DUT (in TOE mode) sends out MSS value of 536 when Peer sends a SYN packet without MSS specified. - 24481: Link bounce observed when T580 adapters are connected to Quanta Switch. This is a Quanta Switch issue. Only one link bounce is seen initially, and thereafter the link remains good with no bounces or errors. - 24197: T580-CR LEDs are OFF in Spider and QSA mode. - 23705: Need a way to read the PHY firmware version of the T5 BT cards. - 21788: Some incoming connections get tunnelled even when an offload server is set up. - 11454: Traffic keeps running even after bringing the VLAN interface down. - 10420: NFS connectivity toggles, while running "iozone -a" on the mounted share This is bug in NFS stack of Linux Kernel, fixed by kernel.org commit 24ca9a8 (SUNRPC: Ensure we return EAGAIN in xs_nospace if congestion is cleared). Workaround: Set toe.toe0_tom.max_host_sndbuf to 2097152 using sysctl to fix the issue. Virtual Function Network (vNIC) ------------------------------- - 43115: [248VF]: ping failure observed on 62nd VF of PF1 after setting VLAN acl using "ip link" command. - 43102: [248VF]: IPv6 ping failure observed over VFs on pinging multiple times. - 41046: Ping fails after VF PCI-e reset. - 36902: [DCBX]: ETS is not supported on vNIC interface. - 36062: Need transparent VLAN encapsulation/de-encapsulation in Hypervisor for SR-IOV Virtual Functions. - 35343: Communication fails over VF interface when both VST and VGT modes are configured. - 31759: soft lockup observed in dmesg while running heavy UDP traffic. - 29916: "tp_tnl_cong_drops" are incrementing while VM receives traffic. - 28561: mps_tcam entry is not getting created intermittently whenever subscribing VF to the multicast group. - 28471: write_coal_fail counters increments after loading vf driver. iWARP RDMA Offload ------------------ - 43785: [RHEL9.3]: NFS/iWARP server hits panic while running iozone from client. - 43775: [RHEL9.3]: Device not responding and system crash seen on starting Open MPI test. - 43619: [6.6.X]: rping/iSER/NVME-oF(RDMA) fails over VLAN interface. - 43233: rping fails to run when iommu is ON. - 40664: "Failed to disconnect RDMA CM connection" seen on client while running perftest tools on RHEL8.2 and Ubuntu 20.04.2 - 40649: [RHEL7.9]: "iwpm_mapping_error_cb" seen while running iwarp traffic with iwpmd enabled. NOTE: This issue is seen only with the inbox rdma-core (rdma-core-22.4-5) of RHEL7.9. The issue can be fixed by re-installing the same or higher version of rdma-core from github. - 40471: LE Hash error observed while running iwarp traffic. - 40469: [Ubuntu-18.04.5]: Cannot run iwarp tools with iwpmd enabled. Workaround: Install rdma-core v28.0 or higher and the issue will not be seen. - 38382: [RHEL7.7/8.1]: "iwpm_mapping_error_cb" seen while running iwarp traffic. NOTE: This is an issue with RHEL 7.7/8.1 distributions and is not seen from RHEL 7.8/8.2. As a workaround, disable iwpmd while running traffic. - 37363: RDMA user-mode fails on RHEL7.6-ALT - 36454: Need DCB (ETS) support for iWARP traffic. - 35622: iozone stalls, while mounting with vers=4 on restarting the NFS server. Workaround: Reduce the grace time and lease time and restart the NFS server. # service nfs stop # echo 10 > /proc/sys/fs/nfs/nlm_grace_period # echo 10 > /proc/fs/nfsd/nfsv4gracetime # echo 10 > /proc/fs/nfsd/nfsv4leasetime # service nfs start - 35540: [RHEL7.6 ARM]: NFS server panics while opening a file in mounted folder from the client. NOTE: This is a kernel issue. Issue not seen with RHEL7.9 kernel installed on the server. - 35105: Need iWARP support for SR-IOV VF. - 19043: [T5]"osu_latency_mt" test failed. Fix: To fix this issue, build MPI with multi-threading support (--enable-threads=multiple) - 5173: Intermittent connection setup failure with NP192 IMB jobs For OpenMPI/RDMA clusters with node counts greater than or equal to 8 nodes, and process counts greater than or equal to 64, you may experience the following RDMA address resolution error when running MPI jobs with the default OpenMPI settings: The RDMA CM returned an event error while attempting to make a connection. This type of error usually indicates a network configuration error. Local host: core96n3.asicdesigners.com Local device: Unknown Error name: RDMA_CM_EVENT_ADDR_ERROR Peer: core96n8 Workaround: Increase the OpenMPI rdma route resolution timeout. The default is 1000, or 1000ms. Increase it to 30000 with this parameter: --mca btl_openib_connect_rdmacm_resolve_timeout 30000 - 4659: OSU benchmark test osu_bibw fails to complete in Platform MPI NOTE: Running osu_bibw benchmark using Platform MPI is known to report issues and is not an iWARP driver bug. iSER ---- - 43619: [6.6.X]: rping/iSER/NVME-oF(RDMA) fails over VLAN interface. - 42166: Errors observed during iSER discovery. NOTE: Login to the target works successfully. - 41130: Issues seen while toggling the interface on multipath enabled lun. - 40690/41004/42084: I/O errors seen while running iser traffic. - 40616: [5.10]: Login fails for more than 9 connections. NOTE: This is a kernel issue and fixed in kernels > 5.10.17. - 38405: "Unknown VPD Code: 0xc9" seen on target during iser login. Workaround: Disable multipathd service on initiator. - 37110: call trace seen within 30 min on iser target machine while running iozone traffic. NOTE: This is an issue with kernel iSER Target module. It is fixed in kernel 5.4 and above. - 34813: Session terminates when running traffic with block size >= 1M. - 32280: Login timeout and IO errors seen, when sg_reset is issued on the LUN from Initiator. - 30607: Trace is seen while clearing target configuration during I/O. WD-UDP ------ - 43233: WD-UDP fails with intel_iommu=on. - 41134: Large number of "tp_tnl_cong_drops" observed. - 40727: Unable to create 63 connections with iperf3. - 39142: wdload needs to handle two cards correctly. - 35482: Applications doing fork are not supported with WD-UDP. - 34201: [T6]: Unable to run more than 512 connections with WD high capacity config file. - 32790: UDP offload fails on receiving end with netserver. Workaround: Start netserver with -f option. NVMe-oF iWARP ------------- - 43805: [RHEL9.3]: Initiator machine hits panic while running iozone test. - 43619: [6.6.X]: rping/iSER/NVME-oF(RDMA) fails over VLAN interface. - 41016: Frequent reconnection issue observed on initiator on running iozone. - 39048: [5.4]: I/O error seen when we format the LUN's. NOTE: This is a kernel issue and is fixed in later Kernel.org kernels. SPDK NVMe-oF iWARP ------------------ - 40481: [SPDK] iozone hang observed while running io against spdk target with interface toggle test. Workaround: Configure in_caps_data (-c) less than 8192 on the target. - 40505: [RHEL8.2]: initiator hits call trace while formatting spdk luns. NOTE: This issue is seen only with ext3. NVMe-oF TOE ----------- - 42327: "Unmaintained driver is detected: NVMe/TCP Target" warning seen in dmesg. - 41626: [NVMe/TCP]: Frequent reconnects seen on initiator on running iozone. - 40852: NVMe/TCP and NVMe/TOE are not supported in RHEL8.2. - 39048: I/O error seen when we format the LUN's. NOTE: This is a kernel issue and is fixed in later Kernel.org kernels. SPDK NVMe-oF TOE ---------------- - 43791: [RHEL9.3/8.9, 6.6.X]: Warning seen on starting nvmf_tgt. - 43181: "failed to connect controller, error 1006" seen when we try to connect to target using connect-all. NOTE: This issue is fixed with with nvme-cli v2.5. - 43156: Starting target errors out with Transport type 'TCP' create failed with both T5 and T6 cards in machine. - 40979: Unable to create targets when "vfio-pci" is used. - 39606: nvmf_tgt reports errors with lvm on nvme disks. NOTE: This is not a Chelsio driver issue. It is a generic issue with SPDK. - 39550: Compilation fails on Virtual Machines. NOTE: This is not a Chelsio driver issue. It is a generic issue with SPDK. To avoid this, ensure that virtual cpu has sse4_1,sse4_2 attributes. Please refer https://mails.dpdk.org/archives/users/2016-February/000283.html SoftiWARP --------- - 41240: [iSER]: Issues observed on Initiator while running iozone traffic with repeated link toggle. - 41004: [iSER]: I/O errors seen on initiator while running iozone traffic. - 40653: Initiator hits panic during nvme discovery. NOTE: This is a kernel issue and fixed in > 5.10.18. - 39340: Trace seen on initiator while running nvme traffic with link toggle. - 39156: Connection errors seen intermittently while running iser traffic. - 39037: I/O errors (WRITE) seen on while running NVMe-oF iozone traffic. Workaround: This is fixed in the latest kernel.org kernels. As a workaround, increase the number of retries for nvme host driver. # modprobe nvme-core max_retries=15 LIO iSCSI Target Offload ------------------------ - 41924: Trace seen while running IO to brd device. - 33602: Connection errors due to dataout_timeout seen while running traffic. iSCSI PDU Offload Target ------------------------ - 36709: Kernel panic seen during stop/start, while running 2 port traffic. - 31814: Connection reset observed while running I/O when iommu is enabled. iSCSI PDU Offload Initiator --------------------------- - 42714: [Debian 11.5]: Kernel panic seen in 1 hour of starting 2 port traffic. - 42330: [RHEL 8.6/9.0]: Connection failure and stuck TID seen during discovery. NOTE: The issue is fixed in upstream and is available from RHEL8.7/9.1. - 42291: [RHEL 9.0]: Persistent login fails after reboot. - 41241/42691: Target stop & start with IO running on luns is not supported. NOTE: The issue is fixed in upstream and is available in Kernels 5.15.35 and above. - 41977: Few node/session commands fail, when logged in over iface. - 38376: connection errors are seen during logout of the LUN. - 32813: Observing few connection errors while running traffic. - 27066: Soft lock-up trace may appear while running heavy traffic. Workaround: Increase the value of 'node.session.xmit_thread_priority' parameter in the Open-iSCSI configuration file (/etc/iscsi/iscsid.conf). Crypto Offload -------------- - 43758: [RHEL9.3]: Ping failure observed on TLS TOE nginx server upon starting ab download connections from clients. - 41332: [Co-processor]: Crypto counters do not increment on nginx server with upload test from PEER. - 41299: [TLS TOE]: Abrupt killing of openssl server with heavy traffic is not supported. - 41209: [TLS TOE]: Client doesn't transfer data when lower MTU is configured on router. - 41074: [TLS TOE]: Only AES128-GCM-SHA256 cipher is supported. - 39969: [Co-processor]: Issues observed while running on dual socket machines. - 39748: OpenVPN is not supported. NOTE: This is not a Chelsio driver issue. The Application does not have kTLS support. - 39666: [Co-processor]: Connections in IPsec tunnel fail to establish if t4_tom is loaded. - 39584: Apache is not supported. NOTE: This is not a Chelsio driver issue. The Application does not have kTLS support. - 38690: [Co-processor]: Need openssh support. - 38677: [Co-processor]: VSFTP is not supported. - 37442: [Co-processor]: Issues with IPsec tunnelling. Workaround: This is a kernel issue and not seen in 5.3 and above kernels. Disable IOMMU and the issue will not be seen. - 35686: Inline TLS is not supported on memory free adapters. - 32884: Machine hits kernel panic when running IPSEC with AES-CTR with larger payload sizes. Note: This is a kernel issue, and it is fixed with http://www.mail-archive.com/linux-crypto@vger.kernel.org/msg25422.html patch. Fix is available in kernels >=4.9.37 - 32889: Running HMAC tests by specifying "-engine af_alg" fails. Workaround: Don't specify "-engine af_alg" since it gets offloaded on to the adapter even without this parameter. Data Center Bridging (DCB) -------------------------- - 39742: dcb_info does not show local App TLVs, when DCBX is disabled on PEER. - 38705: Only few connections are reset, when iSCSI DCB priority is updated in switch. - 37609: Intermittent PFC failures seen. - 37518: Need to show local, remote and operational DCBX parameters in devlog/dcb_info. - 34552: PFC pause frames with priority > 3 are not honoured by T6 adapter. Workaround: Configure priority 0-3 for PFC. - 30002: "Failed DCB Set Application Priority" seen in dmesg if DCBX configuration is updated on Switch. FCoE full offload Initiator --------------------------- - 43699: Warning observed while compiling csiostor in Debian 12.X OS. - 42097: HMA configuration failure seen during PCI bus reset, when csiostor is loaded. - 42085: csiostor reports link down on Port0 when Port1 is shut on switch end. - 41649: Repeated shut and no-shut of initiator switch port is not supported. - 35737: Repeated shut and no-shut of target switch port with less delay is not supported. - 33967: Running FCoE full offload and openfcoe (NIC) initiator traffic simultaneously is not supported. - 34116: Panic observed while loading csiostor after cxgb4 when multiple adapter are present in RHEL7.X platforms. NOTE: This is a kernel issue. - 33948: Need scsi-stats under sysfs. Offload Bonding Driver ---------------------- - 43854: Need support for Ubuntu 22.04.3. - 43789: Connections are not resuming on removing and adding both the slaves to the bond. - 43795: [RHEL9.3, 6.6.X, Debian 12.4]: System hits panic on removing and adding both the slaves to the bond repeatedly using ifenslave with rx traffic. - 43786: [RHEL9.3, 6.6.X]: IPv6 ping fails over the bond. - 36972: No Load balancing in RHEL7.X platforms with mode=2 NOTE: This issue is seen only with iperf and not with other applications like netperf. - 29564: All Rx connections are not resuming immediately on failover. Workaround: disable frto on peer using sysctl - 21419: Offload connections need to be migrated to host stack when last slave is removed from the bond. Offload Multi-Adapter Failover (MAFO) ------------------------------------- - 43854: Need support for Ubuntu 22.04.3. - 41483: Repeated shut and no-shut of switch port with IPv4 bidirectional traffic is not supported. - 41481: Issues with IPv6 failover. - 32092: Traffic fails to run on backup interface if two non-identical cards are used. Workaround: Disable tcp_timestamps using [root@host ~]# sysctl -w net.ipv4.tcp_timestamps=0 UDP Segmentation Offload and Pacing ----------------------------------- - 13872: Application exits immediately when interface goes down. - 13484: Need a way to reset the pacing values to default value. Offload IPv6 ------------ - 7263: IPv6 offload does not support "IPv4-mapped IPv6 addresses". WD Sniffing And Tracing Utility ------------------------------- - 40885: Start/stop of tracer continuously with traffic is not supported. - 40833: tp_tnl_cong_drops seen on tracer while running wd_tcpdump_trace. Workaround: Disable PAUSE on the Chelsio ports of the tracer machine. Classification and Filtering ---------------------------- - 40102: Unable to create hash filters on SO adapter. - 37865: Should report error when creating Hash filter rule with not all tuples provided in filterMask. - 37333: Filter rule does not get honoured when "frag" is set to "1" along with fport or lport tuples. - 35952: filter do not honour when Don't Fragment(DF) and More Fragments(MF) flags are set in a packet. - 35653: Kernel invokes panic when sending traffic to any RspQ ID other than ethernet queue using TCAM filter rule. - 34757: Need support for stateful NAT. - 34040: [VxLAN encap] can not classify packets based on outer header in VxLAN traffic (encap_lookup=0). - 33640: Need support for src-mac match for Filtering. - 33535: Kernel panic seen when enabling tracing to port-1 with hashfilter-config file. - 32677: Need proper error message when hash filter entries get exhausted. - 30260: Filter rule is not honoured for packet with both outer and inner vlan. - 21118: VLAN insertion/extraction/rewrite is not supported with DDR filters. - 10219: Rule based on ethertype 0x8906(FCoE) isn't getting honoured. OVS Offload Driver ------------------ - 37130: LIP should be provided for IPv6 filter to be hit. - 33706: Exact match filters (specifying 4 tuples) with VxLAN currently not supported. - 33662: 2nd VxLAN tunnel with different UDP port won't get offloaded. - 33280: VxLAN VNI rewrite is not supported. - 33271: Hitcounts are not incrementing seamlessly when less than 16 flows are configured. Mesh Topology ------------- - 38780: rping fails to run when iwpmd is enabled. - 38744: Need a method to configure two network interfaces with IPv6 address on same subnet. - 37267: Need a utility to configure network interfaces in mesh topology. Traffic Management ------------------ - 39706: [Class Rate Limiting][25G]: Throughput drop observed while running multiple connections on multiple classes. - 39533: Throughput exceeds the set rate if burst size > 13043. - 35985: No error observed on configuring rate limit more than link speed. - 34124: [TM+TLS]: Higher bandwidth observed for TLS+TOE connections. NOTE: Configure burst-size 16384 to rate limit TLS+TOE connections. - 17207: Class rate limiting with mode flow is not supported for NIC traffic. Unified Boot Software --------------------- - 42108: Link timeout takes more than 5 minutes when peer side link is administratively down. - 40158: Machine keeps rebooting in legacy mode with EFI based OS already present on LUN. - 40457: Need support for iPXE. - 39995: Option ROM loading with adapters having different versions. NOTE: With multiple adapters having different types and versions of Option ROMs, the loading of Option ROM depends on factors like System type (Legacy only, uEFI or dual bios), versions, first device initializing etc. So, it is highly recommended to use the same Option ROM (type and version) on all Chelsio adapters in the system. - 39895: iSCSI boot should allow multiple target and luns to save as boot device. - 36680: Need support for AARCH64 in uEFI driver. - 35708: [Lenovo]: HII is currently not supported on latest uEFI v2.6 machines. Workaround: Use "drvcfg -s" from uEFI shell to configure the adapter. - 23143: [HII] Behaviour on IBM x3850 Gen 6 needs few modifications. ******************************************************************************** Copyright (C) 2024 Chelsio Communications. All Rights Reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Chelsio Communications. Chelsio Communications assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document. Except as permitted by such license, no part of this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means without the express written consent of Chelsio Communications.